import of v0.29.1 (#1865)

Author: whimboo

CHANGES.md

@@ -3,8 +3,58 @@ Change log
 
 All notable changes to this program are documented in this file.
 
+0.29.1  (2021-04-09), `970ef713fe58`)
+-------------------------------------
+
+### Known problems
+
+- _macOS 10.15 (Catalina):_
+
+  Due to the requirement from Apple that all programs must be
+  notarized, geckodriver will not work on Catalina if you manually
+  download it through another notarized program, such as Firefox.
+
+  Whilst we are working on a repackaging fix for this problem, you can
+  find more details on how to work around this issue in the [macOS
+  notarization] section of the documentation.
+
+- _Android:_
+
+  Marionette will only be enabled in GeckoView based applications when the
+  Firefox preference `devtools.debugger.remote-enabled` is set to `True` via
+  [`moz:firefoxOptions`]. This will be fixed in one of the upcoming Firefox
+  for Android releases.
+
+### Added
+
+- When testing GeckoView based applications on Android it's now enough to
+  specify the `androidPackage` capability. The appropriate activity name,
+  and required intent arguments will now automatically be used for
+  applications released by Mozilla.
+
+- Native AArch64 (M1) builds of geckodriver for MacOS are now available. These
+  are currently shipped as Tier2 due to missing test infrastructure. Please let
+  us know if you experience issues.
+
+### Fixed
+
+- Fixed a stack overflow crash in thread 'webdriver dispatcher' when
+  handling certain device errors.
+
+- Fixed an application crash due to missing permissions on unrooted devices
+  by changing the location of the test related files, e.g the profile folder.
+  Therefore the deprecated --android-storage command line argument
+  now defaults to the `sdcard` option, which changed its location to
+  `$EXTERNAL_STORAGE/Android/data/%androidPackage%/files/`. With this change
+  proper support for unrooted devices running Android 10+ has been added.
+
+  _Note_: Do not use the --android-storage command line argument
+  anymore unless there is a strong reason. It will be removed in a future
+  release.
+
+
 0.29.0  (2021-01-14, `cf6956a5ec8e`)
---------------------
+------------------------------------
 
 ### Known problems
 
@@ -18,6 +68,20 @@ All notable changes to this program are documented in this file.
   find more details on how to work around this issue in the [macOS
   notarization] section of the documentation.
 
+- _Android:_
+
+  Marionette will only be enabled in GeckoView based applications when the
+  Firefox preference `devtools.debugger.remote-enabled` is set to `True` via
+  [`moz:firefoxOptions`]. This will be fixed in one of the upcoming Firefox
+  for Android releases.
+
+  In some cases geckodriver could crash due to a stack overflow when handling
+  certain device errors.
+
+  On unrooted Android 10+ devices startup crashes of the application can be
+  experienced due to an inappropriate location of test related files, e.g the
+  profile folder.
+
 ### Added
 
 - Introduced the new boolean capability `moz:debuggerAddress` that can be used
@@ -30,7 +94,7 @@ All notable changes to this program are documented in this file.
   Firefox aka [Fission] will be not available.
 
 0.28.0  (2020-11-03, `c00d2b6acd3f`)
---------------------
+------------------------------------
 
 ### Known problems
 
@@ -44,6 +108,20 @@ All notable changes to this program are documented in this file.
   find more details on how to work around this issue in the [macOS
   notarization] section of the documentation.
 
+- _Android:_
+
+  Marionette will only be enabled in GeckoView based applications when the
+  Firefox preference `devtools.debugger.remote-enabled` is set to `True` via
+  [`moz:firefoxOptions`]. This will be fixed in one of the upcoming Firefox
+  for Android releases.
+
+  In some cases geckodriver could crash due to a stack overflow when handling
+  certain device errors.
+
+  On unrooted Android 10+ devices startup crashes of the application can be
+  experienced due to an inappropriate location of test related files, e.g the
+  profile folder.
+
 ### Added
 
 - The command line flag `--android-storage` has been added, to allow geckodriver
@@ -66,7 +144,7 @@ All notable changes to this program are documented in this file.
   and querying its attributes are no longer needed, and have been removed.
 
 0.27.0  (2020-07-27, `7b8c4f32cdde`)
---------------------
+------------------------------------
 
 ### Security Fixes
 
@@ -92,10 +170,20 @@ All notable changes to this program are documented in this file.
   find more details on how to work around this issue in the [macOS
   notarization] section of the documentation.
 
+- _Android:_
+
+  Marionette will only be enabled in GeckoView based applications when the
+  Firefox preference `devtools.debugger.remote-enabled` is set to `True` via
+  [`moz:firefoxOptions`]. This will be fixed in one of the upcoming Firefox
+  for Android releases.
+
+  In some cases geckodriver could crash due to a stack overflow when handling
+  certain device errors.
+
 ### Added
 
 - To set environment variables for the launched Firefox for Android,
-  it is now possible to add an `env` object on `moz:firefoxOptions`
+  it is now possible to add an `env` object on [`moz:firefoxOptions`]
   (note: this is not supported for Firefox Desktop)
 
 - Support for print-to-PDF
@@ -126,7 +214,7 @@ All notable changes to this program are documented in this file.
 - Windows and Linux binaries are again statically linked.
 
 0.26.0  (2019-10-12, `e9783a644016'`)
-------------------------------------
+-------------------------------------
 
 Note that with this release the minimum recommended Firefox version
 has changed to Firefox ≥60.
@@ -149,6 +237,16 @@ has changed to Firefox ≥60.
   runtime] installed on your system for the binary to run.  This
   is a known bug which we weren't able fix for this release.
 
+- _Android:_
+
+  Marionette will only be enabled in GeckoView based applications when the
+  Firefox preference `devtools.debugger.remote-enabled` is set to `True` via
+  [`moz:firefoxOptions`]. This will be fixed in one of the upcoming Firefox
+  for Android releases.
+
+  In some cases geckodriver could crash due to a stack overflow when handling
+  certain device errors.
+
 ### Added
 
 - Support for Firefox on Android
@@ -522,7 +620,7 @@ Firefox and Selenium versions have changed:
 
 - Firefox will now be started with the `-foreground` and `-no-remote`
   flags if they have not already been specified by the user in
-  `moz:firefoxOptions`.
+  [`moz:firefoxOptions`].
 
   `-foreground` will ensure the application window gets focus when
   Firefox is started, and `-no-remote` will prevent remote commands
@@ -732,7 +830,7 @@ Note that with geckodriver 0.19.0 the following versions are recommended:
 
 - To pick up a prepared profile from the filesystem, it is now possible
   to pass `["-profile", "/path/to/profile"]` in the `args` array on
-  `moz:firefoxOptions`
+  [`moz:firefoxOptions`]
 
 - geckodriver now recommends Firefox 53 and greater
 
@@ -882,7 +980,7 @@ and greater.
 - Fix broken unmarshaling of _Get Timeouts_ response format from Firefox
   52 and earlier (fixed by [Jason Juang])
 
-- Allow preferences in `moz:firefoxOptions` to be both positive- and
+- Allow preferences in [`moz:firefoxOptions`] to be both positive- and
   negative integers (fixed by [Jason Juang])
 
 - Allow IPv6 hostnames in the proxy configuration object
@@ -1035,7 +1133,7 @@ and greater.
   `/session/{sessionId}/moz/xbl/{elementId}/anonymous_by_attribute` to
   return an anonymous element by a name and attribute query
 
-- Introduced a `moz:firefoxOptions` capability to customise a Firefox
+- Introduced a [`moz:firefoxOptions`] capability to customise a Firefox
   session:
 
   - The `binary`, `args`, and `profile` entries on this dictionary
@@ -1054,7 +1152,7 @@ and greater.
 ### Changed
 
 - `firefox_binary`, `firefox_args`, and `firefox_profile` capabilities
-  removed in favour of the `moz:firefoxOptions` dictionary detailed above
+  removed in favour of the [`moz:firefoxOptions`] dictionary detailed above
   and in the [README]
 
 - Removed `--no-e10s` flag, and geckodriver will from now rely on the

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "geckodriver"
-version = "0.29.0"
+version = "0.29.1"
 description = "Proxy for using WebDriver clients to interact with Gecko-based browsers."
 keywords = ["webdriver", "w3c", "httpd", "mozilla", "firefox"]
 repository = "https://hg.mozilla.org/mozilla-central/file/tip/testing/geckodriver"
@@ -17,8 +17,8 @@ hyper = "0.13"
 lazy_static = "1.0"
 log = { version = "0.4", features = ["std"] }
 marionette = { path = "./marionette" }
-mozdevice = "0.3.1"
-mozprofile = "0.7.1"
+mozdevice = "0.3.2"
+mozprofile = "0.7.2"
 mozrunner = "0.12.1"
 mozversion = "0.4.1"
 regex = { version="1.0", default-features = false, features = ["perf", "std"] }
@@ -27,7 +27,7 @@ serde_derive = "1.0"
 serde_json = "1.0"
 serde_yaml = "0.8"
 uuid = { version = "0.8", features = ["v4"] }
-webdriver = "0.43.0"
+webdriver = "0.43.1"
 zip = { version = "0.4", default-features = false, features = ["deflate"] }
 
 [[bin]]

doc/Flags.md

@@ -3,6 +3,11 @@ Flags
 
 #### <code>&#x2D;&#x2D;android-storage <var>ANDROID_STORAGE</var></code>
 
+**Deprecation warning**: This argument is deprecated and planned to be removed
+with the 0.31.0 release of geckodriver. As such it shouldn't be used with version
+0.30.0 or later anymore. By default the automatic detection will now use the
+external storage location, which is always readable and writeable.
+
 Selects the test data location on the Android device, eg. the Firefox profile.
 By default `auto` is used.
 
@@ -40,10 +45,9 @@ By default `auto` is used.
     shell user. Commands will be executed via <code>su</code>.
  <tr>
   <td>sdcard
-  <td><p>Location: <code>/mnt/sdcard/test_root</code></p>
-    This location is not supported on Android 11+ due to the
-    <a href="https://developer.android.com/about/versions/11/privacy/storage">
-    changes related to scoped storage</a>.
+  <td><p>Location: <code>$EXTERNAL_STORAGE/Android/data/%androidPackage%/files/test_root</code></p>
+    This location is supported by all versions of Android whether if the device
+    is rooted or not.
 </table>
 
 
@@ -135,6 +139,33 @@ system to atomically assign a free port.
 Attach [browser toolbox] debugger when Firefox starts.  This is
 useful for debugging [Marionette] internals.
 
+To be prompted at the start of the test run or between tests,
+you can set the `marionette.debugging.clicktostart` preference to
+`true`.
+
+For reference, below is the list of preferences that enables the
+chrome debugger. These are all set implicitly when the
+argument is passed to geckodriver.
+
+  * `devtools.browsertoolbox.panel` -> `jsdebugger`
+
+    Selects the Debugger panel by default.
+
+  * `devtools.chrome.enabled` → true
+
+    Enables debugging of chrome code.
+
+  * `devtools.debugger.prompt-connection` → false
+
+    Controls the remote connection prompt.  Note that this will
+    automatically expose your Firefox instance to localhost.
+
+  * `devtools.debugger.remote-enabled` → true
+
+    Allows a remote debugger to connect, which is necessary for
+    debugging chrome code.
+
+
 [browser toolbox]: https://developer.mozilla.org/en-US/docs/Tools/Browser_Toolbox
 
 

doc/Releasing.md

@@ -60,9 +60,9 @@ If a feature was added but removed before release, there is no reason
 to list it as a change.
 
 It is good practice to also include relevant information from the
-[webdriver] and [rust-mozrunner] crates, since these are the two most
-important dependencies of geckodriver and a lot of its functionality
-is implemented there.
+[webdriver], [rust-mozrunner], and [rust-mozdevice] crates, since these
+are the most important dependencies of geckodriver and a lot of its
+functionality is implemented there.
 
 We follow the writing style of the existing change log, with
 one section per version (with a release date), with subsections
@@ -73,8 +73,9 @@ to make the file readable in a text editor as well as rendered HTML.
 fmt(1) does a splendid job at text formatting.
 
 [CHANGES.md]: https://searchfox.org/mozilla-central/source/testing/geckodriver/CHANGES.md
+[webdriver]: https://searchfox.org/mozilla-central/source/testing/webdriver
 [rust-mozrunner]: https://searchfox.org/mozilla-central/source/testing/mozbase/rust/mozrunner
-
+[rust-mozdevice]: https://searchfox.org/mozilla-central/source/testing/mozbase/rust/mozdevice
 
 Update libraries
 ----------------

doc/Support.md

@@ -22,7 +22,11 @@ and required versions of Selenium and Firefox:
     <th>max
   </tr>
  </thead>
-
+ <tr>
+  <td>0.29.1
+  <td>≥ 3.11 (3.14 Python)
+  <td>60
+  <td>n/a
  <tr>
   <td>0.29.0
   <td>≥ 3.11 (3.14 Python)

src/android.rs

@@ -171,7 +171,18 @@ impl AndroidHandler {
                 buf
             }
             AndroidStorage::Internal => PathBuf::from("/data/local/tmp/test_root"),
-            AndroidStorage::Sdcard => PathBuf::from("/mnt/sdcard/test_root"),
+            AndroidStorage::Sdcard => {
+                // We need to push the profile to a location on the device that can also
+                // be read and write by the application, and works for unrooted devices.
+                // The only location that meets this criteria is under:
+                //     $EXTERNAL_STORAGE/Android/data/%options.package%/files
+                let response = device.execute_host_shell_command("echo $EXTERNAL_STORAGE")?;
+                let mut buf = PathBuf::from(response.trim_end_matches('\n'));
+                buf.push("Android/data");
+                buf.push(&options.package);
+                buf.push("files/test_root");
+                buf
+            }
         };
 
         debug!(
@@ -431,7 +442,19 @@ mod test {
                 buf
             }
             AndroidStorage::Internal => PathBuf::from("/data/local/tmp/test_root"),
-            AndroidStorage::Sdcard => PathBuf::from("/mnt/sdcard/test_root"),
+            AndroidStorage::Sdcard => {
+                let response = handler
+                    .process
+                    .device
+                    .execute_host_shell_command("echo $EXTERNAL_STORAGE")
+                    .unwrap();
+
+                let mut buf = PathBuf::from(response.trim_end_matches('\n'));
+                buf.push("Android/data/");
+                buf.push(&package);
+                buf.push("files/test_root");
+                buf
+            }
         };
         assert_eq!(handler.test_root, test_root);